A Hybrid Model for Detection and Elimination of Near- Duplicates Based on Web Provenance for Effective Web Search

Users of World Wide Web utilize search engines for information retrieval in web as search engines play a vital role in finding information on the web. But, the voluminous amount of web documents has weakened the performance and reliability of web search engines. As, the subsistence of near-duplicate data is an issue that accompanies the growing need to incorporate heterogeneous data. These pages either increase the index storage space or increase the serving costs thereby irritating the users. Near-duplicate detection has been recognized as an important one in the field of plagiarism detection, spam detection and in focused web crawling scenarios. Such near-duplicates can be detected and eliminated using the concept of Web Provenance and TDW matrix Algorithm. The proposed work is the model that combines content, context, semantic structure and trust based factors for classifying and eliminating the results as original or near-duplicates.